Redundant automation system and method for operating the redundant automation system

ABSTRACT

A redundant automation system and a method for operating the redundant automation system which is provided with a first subsystem and a second subsystem that each process a control program while controlling a technical process, one of these subsystems operating as a master and the other subsystem operating as a slave, and the slave assuming the function of the master if the master fails such that it becomes possible to dispense with temporally synchronous communication between the participants with regard to the synchronization of the program processing in the two subsystems, thus reducing the communication load.

BACKGROUND OF THE INVENTION

1. Field of the Invention

The invention relates to a redundant automation system and method foroperating the redundant automation system which is provided with a firstsubsystem and a second subsystem, where the subsystems each redundantlyprocess a control program while controlling a technical process, and oneof these subsystems operate as a master and the other subsystem operatesas a slave, and the slave assumes the function of the master if themaster fails.

2. Description of the Related Art

In the automation environment, there is an increasing demand for highlyavailable solutions (H systems) that are suitable for minimizingpossible downtimes of an installation. The development of such highlyavailable solutions is very cost-intensive, where an H system usuallyused in the automation environment is distinguished by the fact that twoor more subsystems in the form of automation devices or computer systemsare coupled to one another via a synchronization connection. Inprinciple, both subsystems can have read and/or write access toperipheral units connected to this H system. One of the two subsystemsleads with respect to the peripherals connected to the system. Thismeans that outputs to peripheral units or output information for theseperipheral units is/are effected only by one of the two subsystems whichoperates as a master or has assumed the master function. So that bothsubsystems can run in a synchronous manner, the subsystems aresynchronized at regular intervals via the synchronization connection.With respect to the frequency and extent of synchronization, differentforms may be distinguished (e.g., warm standby, hot standby).

An H system often requires a smooth “failover”, if one of the subsystemsfails and it is necessary to change over to the other subsystem. Thismeans that, despite this unplanned changeover or this unplanned changefrom one subsystem to the other, this changeover or change does not havea disruptive effect on the technical process to be controlled. Here, itis permissible for a (short) dead time, during which the outputs remainat their last valid process output values, to occur at the outputs ofthe connected peripherals. However, a jump (surge) in the values atthese outputs on account of the changeover is undesirable and shouldtherefore be avoided. Therefore, “smooth” should also be understood asmeaning the continuity of the curve shape of the process output values.

In order to achieve this smoothness, the two subsystems must have thesame system state at the time of the failure. This is ensured by thesuitable synchronization method. If both subsystems are processing theinput information (inputs) of the process, both systems are in the samesystem state when they change their respective “thread global” data(shared data of programs, i.e., programs with different priorities) inthe same manner given the same process input data or process inputinformation. In order to achieve this, the synchronization methodensures that the individual threads of the two subsystems areinterrupted or executed in the same manner. This results in an identical“thread mountain”.

The Siemens catalog ST 70, chapter 6, 2011 edition, discloses aredundant automation system that consists of two subsystems and isintended to increase the availability of an installation to becontrolled. For this purpose, the automation system is provided withmeans that initially decide, based on an event, which program must bestarted in order to suitably react to the event. If, for example, duringthe execution of a program, an event in the form of a pending alarm forthe technical process to be controlled is applied to a signaling inputof the automation system, the running program is usually stopped at awaiting point and a program that is intended to analyze the alarm andinitiate measures that eliminate the cause of the alarm is started. Thisautomation system is regularly synchronized, and it is ensured that thefailure of one of these subsystems does not have a disruptive effect ona process to be controlled because the other subsystem can continue theexecution or processing of the corresponding part of its respectivecontrol program or the execution or processing of the correspondingparts of this control program.

If, for example, an event that has occurred in a first subsystem is notsynchronized with a second subsystem of an automation system comprisingtwo subsystems and, after the event has been processed by the firstsubsystem, this subsystem fails, the course of a technical process to becontrolled may be disrupted. This is because the second subsystem(without knowledge of the event) runs through a different program path,representing the execution order of the programs, from the program paththat would be run through by the second subsystem with knowledge of theevent and that would also be necessary to avoid disrupting the course ofthe technical process to be controlled.

In this context, it should be noted that a program is understood asmeaning both a program as such and a subroutine, a part of a program, atask, a thread, an organizational module, a functional module or anothersuitable program code for implementing an automation function, where theprograms of an automation system usually are categorized into priorityclasses and are processed or executed according to their associatedpriority.

EP 0 907 912 B1 discloses a synchronization method for an automationsystem constructed from two subsystems. This synchronization method isbased on temporally synchronous coupling of the two subsystems, bothsubsystems waiting for a response from the respective other participantat suitable program points at which adjustment is intended and only thenrespectively continuing their program processing in a temporallysynchronous manner. The disadvantage is the long waiting times beforereceiving the responses needed for the temporal synchronization.

US 2002/0095221 A1 describes a redundant automation system provided witha first controller and a second controller. Suitable measures that makeit possible to execute periodic tasks in a timely manner are provided.

SUMMARY OF THE INVENTION

It is therefore an object of the invention to provide a redundantautomation system and method which makes it possible to dispense withtemporally synchronous communication between the participants withregard to the synchronization of the program processing in the twosubsystems.

This and other objects and advantages are achieved in accordance withthe invention by an automation system and method in which the masteradvantageously does not have to (actively) wait for a response from theslave to be able to continue its program processing. That is, allrelevant information is transmitted from the master to the slave in atemporally asynchronous manner. As a result, the processing performanceof the master is decoupled from the communication bandwidth availablefor event synchronization, which is important, in particular, withregard to the increasing imbalance between the increase in theprocessing performance of the processors, on the one hand, and theincrease in the communication performance, on the other hand. This isbecause the communication performance usually cannot keep up with theincreasing processing performance.

After an event has occurred, the two subsystems are synchronized in asynchronous manner such that both the master and the slave run throughthe same program paths on account of this event, the runs being effectedin a temporally asynchronous manner. Thus, the master temporally leadsthe slave or the slave temporally trails the master with regard to theprogram processing. In this context, “trailing” or “leading” isunderstood as meaning the time difference between the beginning of theprocessing of the processing sections by the master and the beginning ofthe processing of the processing sections by the slave, whichcorresponds to the time at which the release signal occurs.

On account of the temporally asynchronous communication between themaster and the slave, it is also possible to use slow communicationconnections for setting up a highly available automation system. Thus,it is also possible to provide a communication connection that is poorper se with regard to the transmission bandwidth or response time orelse a communication connection that is also used by other communicationparticipants and is thus not exclusively available to the twoparticipants for synchronization purposes. It is therefore possible todispense with a separate synchronization connection. Furthermore, largedistances between the two participants may also be overcome withoutimpairing the system performance too much as a result of long signalpropagation times or long latency times.

In one embodiment of the invention, the master is used to also transmitprocess input values to the slave at the time at which the currentreleases are transmitted. The information relevant to the slave isinitially combined or collected and is finally transmitted to the slave.In contrast with known temporal synchronization methods, during whichrelevant information must be immediately transmitted to the slave, thismeans a considerably reduced amount of “management effort” both for themaster and for the slave or the reserve.

In another embodiment of the invention, the slave acknowledges therespective release to the master after the respective processingsections have been processed. The number of unacknowledged releasesmakes the master aware of the current trailing of the slave, as a resultof which the master can take suitable measures to prevent the temporaltrailing from becoming too great.

Other objects and features of the present invention will become apparentfrom the following detailed description considered in conjunction withthe accompanying drawings. It is to be understood, however, that thedrawings are designed solely for purposes of illustration and not as adefinition of the limits of the invention, for which reference should bemade to the appended claims. It should be further understood that thedrawings are not necessarily drawn to scale and that, unless otherwiseindicated, they are merely intended to conceptually illustrate thestructures and procedures described herein.

BRIEF DESCRIPTION OF THE DRAWINGS

The invention, its refinements and advantages are explained in moredetail below using the drawing which illustrates an exemplary embodimentof the invention and in which:

FIGS. 1 to 3 show sequences of temporally asynchronous coupling of twosubsystems;

FIG. 4 shows a redundant automation system; and

FIG. 5 is a flowchart of the method in accordance with the invention.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

The same parts in FIGS. 1 to 4 are provided with the same referencesymbols.

Reference is initially made to FIG. 4 which illustrates a redundantautomation system that is known per se and comprises two subsystems. Afirst subsystem Ta and a second subsystem Tb are connected to aperipheral unit Pe via a field bus Fb. Here, the field bus Fb complieswith the PROFIBUS-DP specification, for example.

In principle, other bus systems, such as Ethernet, Fieldbus, Modbus orparallel bus systems, are also suitable. The peripheral unit Pereceives, via input lines Es, signals from transducers or measuringtransducers, which are used to detect a process state, and outputs, viaoutput lines As, signals to actuators that are used to influence theprocess. The process as well as the transducers, measuring transducersand actuators are not illustrated in the figure for purposes of clarity.The two subsystems Ta, Tb execute the same control program in a cyclicaland synchronous manner. A synchronization connection Sv is provided tosynchronize the subsystems, the redundancy and monitoring functionsbeing implemented via this synchronization connection Sv.

In order to explain event-synchronous processing of the controlprograms, reference is made below to FIGS. 1 to 3 that illustratesequences of temporally asynchronous coupling of two subsystems. In thiscontext, “event-synchronous processing” means that both the master andthe slave run through the same program paths of the respective controlprogram on account of an event, the runs being effected in a temporallyasynchronous manner.

It is assumed that one subsystem is operated as a master M and onesubsystem is operated as a slave S or a reserve. The master M thereforeleads with respect to the control of a technical process and undertakesprocess control, the master reading the process input information orprocess input values from the peripheral unit Pe (FIG. 4) and makingit/them available to the slave S in a temporally asynchronous manner.The slave S assumes the master function or the role of master only ifthe master M fails on account of a fault.

The master M processes a program P1 for controlling the technicalprocess, the slave S also processing a program P2 corresponding to thiscontrol program P1. Both control programs P1, P2 have a multiplicity ofprocessing sections (Va) of different duration, the control programs P1,P2 being able to be interrupted at the respective beginning and therespective end of each processing section Va. The beginning and end ofeach processing section Va, which usually comprises a multiplicity ofprogram codes, therefore represent interruptible program points orbreakpoints 0, 1, 2, . . . y. If necessary, the respective controlprogram P1, P2 can be interrupted at these points 0, 1, 2, . . . y usingthe master M and the slave S in order to be able to initiate suitablereactions after an event or a process alarm has occurred. Furthermore,the respective control program P1, P2 can be interrupted at thesebreakpoints 0, 1, 2, . . . y so that the master M and the slave S caninterchange releases, acknowledgements or other information via thefield bus Fb or via the synchronization connection Sv (FIG. 4). After arespective predefinable or predefined interval of time Zi, i=1, 2, . . .y has expired and at the respective time at which a breakpoint followingthe expiry of the respective interval of time Zi occurs, preferably thefirst breakpoint following the respective interval of time Zi, themaster M transmits a release or release signal to the slave S, whichrelease or release signal indicates to the slave S the processingsection Va up to which the slave S can process the control program P2.These processing sections Va of the control program P2 correspond tothose that have already been processed by the master M during theprocessing of the control program P1. In the present exemplaryembodiment, it is assumed that, after an interval of time Z1 hasexpired, the master M transmits a release F1 to the slave S at a time t1and at a time t2 at which a first breakpoint P1_6 (breakpoint 6) followsthe interval of time Z1. This release F1 comprises the information forthe slave S indicating that the latter can process its control programP2 to be processed up to a breakpoint P2_6 (breakpoint 6), thebreakpoint P2_6 of the control program P2 corresponding to thebreakpoint P1_6 of the control program P1. This means that, based on therelease, the slave S can process those processing sections Va of thecontrol program P2 that correspond to the processing sections Va of thecontrol program P1 up to the time at which the release or the releasesignal is generated, in which case it is assumed in the example for thesake of simplicity that the time at which the release is generatedcorresponds to the time at which the release is transmitted to the slaveS. These processing sections Va are therefore processed using the slaveS in a temporally asynchronous manner with respect to the processing ofthe corresponding processing sections Va using the master M, the slave Sprocessing further processing sections Va, after the processing sectionsVa of the control program P2 have been processed by the slave S, onlywhen the master M transmits a further release to the slave S. The timeat which this breakpoint P1_6, P2_6 (breakpoint 6) occurs represents thebeginning of an interval of time Z2 following the interval of time Z1.

The further temporally asynchronous processing of the control programsP1, P2 is performed in the described manner. At a time t3 at which afirst breakpoint P1_A occurs after the expiry of the interval of timeZ2, the master M transmits a further release F2 to the slave S, whichrelease indicates to the slave S that the latter can process furtherprocessing sections Va up to the breakpoint P2_A. These processingsections Va again correspond to those which have already been processedby the master M from the time t2 to the time t3, i.e., up to thebreakpoint P1_A. This means that the slave S processes the processingsections Va from the time t2 of the previous release F1 to the time t3of the current release F2. The time t3 at which the first breakpointP1_A has occurred after the expiry of the interval of time Z2 is thebeginning of an interval of time Z3 following the interval of time Z2.

An event, such as an event comprising a process alarm, may now occurduring an interval of time. In the exemplary embodiment, E is used todenote such an event to which the master M must react in a suitablemanner during the interval of time Z3 at a time t4 in accordance withthe control program P1. In this case, the master M does not transmit arelease F3 to the slave S at a time at which a breakpoint following theinterval of time Z3 occurs after the interval of time Z3 but rather at atime t5 at which a breakpoint P1_C (breakpoint C) following theoccurrence of the event E occurs. This means that the interval of timeZ3 is shortened on account of the event E, the time t5 being thebeginning of a following interval of time Z4. Based on the release F3transmitted to the slave S, the slave S processes those processingsections Va of the control program P2 that correspond to thoseprocessing sections Va of the control program P1 that have already beenprocessed by the master M between the times t3 and t5.

On account of the event E, the master M processes higher-priorityprocessing sections Va during the interval of time Z4, for example, themaster M performs a thread change at the time t5 and, after the intervalof time Z4 has expired at the time t6, again transmits a release F4 at atime t7 at which a first breakpoint P1_12 (breakpoint 12) following theinterval of time Z4 occurs. Based on this release, the slave S likewiseprocesses processing sections Va up to a breakpoint P2_12 (breakpoint12) in the control program P2, these processing sections Vacorresponding to the processing sections Va of the control program P1between the times t5 and t7, and the slave S likewise perform a threadchange.

As explained, the releases from the master M make it possible for theslave S to run through the same “thread mountain” as the master M, whichmeans that the slave S performs a “thread change” at a point in thecontrol program P2 corresponding to the point in the control program P1.The slave S continues its processing only when requested to do so by themaster M by means of a release. With regard to the processing of theprocessing sections, the master M processes them in real time like instand-alone operation or in non-redundant operation and issues releasesfor corresponding processing sections to be processed by the slave S atregular intervals of time and after the occurrence of events, the masterM continuing to process its control program P1 and not actively waitingfor a response from the slave S. With regard to the processing of thecorresponding processing sections, the slave S trails the master M andprocesses the sections based on the issued master releases.

Reference is made below to FIG. 2 which illustrates a transition of therole of master from the master M to the slave S. The master M transmitsreleases F5, F6, F7 to the slave S in the described manner, in whichcase it is assumed that the master M fails at a time t8.

Based on the releases F5 to F7, the slave S processes the processingsections Va of a control program P4 up to a breakpoint P4_B (breakpointB), these processing sections Va corresponding to those processingsections Va of a control program P3 that have been processed using themaster M up to the breakpoint P3_B (breakpoint B).

At times te1, te2, the master M has read access to the peripheral unitPe within the scope of the processing of the control program P3, whichmeans that the master M reads in process input values Ew1, Ew2,processes them in accordance with the control program P3 and generatesprocess output values Aw1, Aw2 that are transmitted to the peripheralunit Pe at times ta1, ta2 by the master M. The master M transmits theprocess input values Ew1, Ew2 to the slave S, which is indicated in thedrawing by curved lines L1, L2. The transmission is effected togetherwith the releases F5, F7 to avoid increasing the communication loadbetween the master M and the slave S while processing the processingsections Va up to these releases F5, F7. The slave S likewise processesthese process input values Ew1, Ew2 in accordance with the controlprogram P4 and likewise generates the process output values Aw1, Aw2that are transmitted to the peripheral unit Pe by the slave S. In thiscase, it is assumed that the peripheral unit Pe is a “switched”peripheral unit having a primary connection and a secondary connection.The primary connection is intended to receive the process output valuesfrom the master M and the secondary connection is intended to receivethe process output values from the slave S, the slave S changing overthe peripheral unit from the primary connection to the secondaryconnection if the slave S detects that the master M has failed.

As explained, it is assumed that the master M fails at a time t8. Theslave S detects the failure, for example, by virtue of the fact that themaster M has not transmitted any sign of life to the slave S via thesynchronization connection Sv or the field bus Fb (FIG. 4) during apredefined duration. After the slave S has detected the failure, such asat a time t9, the slave S does not immediately assume the role ofmaster; this is because the system state of the slave S differs fromthat of the master M at this time t9 and a smooth change or transitionis therefore impossible. At this time t9, the slave S has only processedthe processing sections Va up to a breakpoint P4_6 (breakpoint 6) andthe corresponding processing sections Va of the master M up to abreakpoint P3_6 (breakpoint 6) therefore “lie” in the past. Only after atransition, i.e., after the slave S has processed the processingsections Va released using the release F7 up to the breakpoint P4_B at atime t10, does the slave S assume the role of master and thus thecontrol of the technical process, the slave S changing over theperipheral unit from the primary connection to the secondary connectionat this time t10. During this transition, the (previous) slave Stherefore still runs through the same thread mountain with pathsynchronization and processes the same process input values as thoseprocessed by the (previous) master M before its failure, the (previous)slave S determining the same process output values as the (previous)master M based on these input values. The transition is ended when theaim of the last release (the processing of the processing sections Va upto the breakpoint P4_B in the present example) has been achieved.

The duration of the transition substantially corresponds to the durationof the temporal trailing at the “failover” time. In order to keep thetemporal trailing at a tolerable degree, every release F8 to F12 (FIG.3) from the master M is then acknowledged by the slave S in anasynchronous manner using respective acknowledgements Q8 to Q12 if theslave S has concluded the respective processing. The master M evaluatesthe number of unacknowledged releases and determines the currenttrailing of the slave S therefrom. If the temporal trailing is too highor too long, which may result in a loss of redundancy, for example, themaster M takes suitable measures to reduce the temporal trailing or toavoid it becoming too large.

For example, the master M can suspend or delay the processing oflow-priority threads in response to excessive trailing, the processingof the higher-priority threads requiring considerably less than 100% ofthe computation time. The master M therefore has fewer processingsections to run through and generates fewer releases, with the resultthat the slave S or the reserve can “catch up”.

FIG. 5 is a flowchart of a method for operating a redundant automationsystem provided with a first subsystem and a second subsystem, where thesubsystems each process a control program while controlling a technicalprocess, where one of the subsystems operates as a master and another ofthe subsystems operates as a slave which functions as the master if themaster fails. The method comprises processing sections of the controlprogram to be processed using the master, as indicated in step 510. Arelease is then transmitted from the master to the slave in either aftera predefined interval of time has expired or after an event hasoccurred, as indicated in step 520. Next, those processing sections ofthe control program to be processed are processed, based on the release,using the slave that corresponds to those processing sections of thecontrol program to be processed using the master which has already beenprocessed up to the release, as indicated in step 530.

While there have been shown, described and pointed out fundamental novelfeatures of the invention as applied to a preferred embodiment thereof,it will be understood that various omissions and substitutions andchanges in the form and details of the methods described and the devicesillustrated, and in their operation, may be made by those skilled in theart without departing from the spirit of the invention. For example, itis expressly intended that all combinations of those elements and/ormethod steps which perform substantially the same function insubstantially the same way to achieve the same results are within thescope of the invention. Moreover, it should be recognized thatstructures and/or elements and/or method steps shown and/or described inconnection with any disclosed form or embodiment of the invention may beincorporated in any other disclosed or described or suggested form orembodiment as a general matter of design choice. It is the intention,therefore, to be limited only as indicated by the scope of the claimsappended hereto.

What is claimed is:
 1. A method for operating a redundant automationsystem provided with a first subsystem and a second subsystem, whichsubsystems each process a control program while controlling a technicalprocess, one of said subsystems operating as a master and another ofsaid subsystems operating as a slave which functions as the master ifthe master fails, the method comprising the steps of: processingsections of the control program to be processed using the master;transmitting from the master a release to the slave via one of (i) afield bus and (ii) a synchronization connection in each case one of (i)after a predefined interval of time has expired and (ii) after an eventhas occurred; and processing, based on the release, those processingsections of the control program to be processed in a temporallyasynchronous manner using the slave which corresponds to thoseprocessing sections of the control program to be processed using themaster which has already been processed up to the release.
 2. The methodas claimed in claim 1, wherein the slave acknowledges a respectiverelease to the master after respective processing sections have beenprocessed.
 3. The method as claimed in claim 2, further comprising thesteps of: providing breakpoints at a beginning and an end of arespective processing section; providing one of (a) a time at which abreakpoint following a respective interval of time occurs and (b) a timeat which a breakpoint following an occurrence of an event occurs as abeginning of an interval of time following the respective interval oftime; and transmitting the release to the slave at the time of (a) and(b).
 4. The method as claimed in claim 1, further comprising the stepsof: providing breakpoints at a beginning and an end of a respectiveprocessing section; providing one of (a) a time at which a breakpointfollowing a respective interval of time occurs and (b) a time at which abreakpoint following an occurrence of an event occurs as a beginning ofan interval of time following the respective interval of time; andtransmitting the release to the slave at the time of (a) and (b).
 5. Themethod as claimed in claim 1, further comprising the step of:transmitting process input values from the master to the slave at a timeat which releases are transmitted.
 6. A redundant automation systemincluding a first subsystem and a second subsystem, which subsystemseach process a control program while controlling a technical process,one of said subsystems operating as a master and another of saidsubsystem operating as a slave which functions as the master if themaster fails; wherein the master is configured to process processingsections of the control program to be processed using the master and totransmit a release to the slave via one of (i) a field bus and (ii) asynchronization connection in each case one of (i) after a predefinedinterval of time has expired and (ii) after an event has occurred; andwherein the slave is configured to process based on the release,processing sections of the control program to be processed in atemporally asynchronous manner using the slave which correspond to thoseprocessing sections of the control program to be processed by the masterwhich have already been processed up to the release.
 7. The redundantautomation system as claimed in claim 6, wherein the slave is furtherconfigured to acknowledge a respective release to the master afterrespective processing sections have been processed.
 8. The redundantautomation system as claimed in claim 7, wherein the automation systemis configured to provide breakpoints control programs to be processedusing the master and using the slave in each case at a beginning and atan end of a respective processing section; wherein the master isconfigured to provide one of (a) a time at which a breakpoint followinga respective interval of time occurs and (b) a time at which abreakpoint following an occurrence of an event occurs at the beginningof the respective interval of time following the respective interval oftime; and wherein the master is further configured to transmit therelease to the slave at the time of one of (a) and (b).
 9. The redundantautomation system as claimed in claim 6, wherein the automation systemis configured to provide breakpoints control programs to be processedusing the master and using the slave in each case at a beginning and atan end of a respective processing section; wherein the master isconfigured to provide one of (a) a time at which a breakpoint followinga respective interval of time occurs and (b) a time at which abreakpoint following an occurrence of an event occurs at the beginningof the respective interval of time following the respective interval oftime; and wherein the master is further configured to transmit therelease to the slave at the time of one of (a) and (b).
 10. Theredundant automation system as claimed in claim 6, wherein the master isfurther configured to transmit process input values to the slave at atime at which releases are transmitted.
 11. A slave for a redundantautomation system having a master and another slave, the master and theslave each processing a control program while controlling a technicalprocess, and the slave functioning as the master if the master fails;wherein the slave is configured to process, based on a releasetransmitted via one of (i) a field bus and (ii) a synchronizationconnection by the master processing sections of the control program tobe processed using the slave which correspond to those processingsections of the control program to be processed by the master which havealready been processed up to the release; and wherein the slave isfurther configured to process further processing sections of its controlprogram in a temporally asynchronous manner only if the master transmitsa further release to the slave after this processing.
 12. A master for aredundant automation system having a slave and the master, the masterand the slave each processing a control program while controlling atechnical process, and the slave functioning as the master if the masterfails; wherein the master is configured to process processing sectionsof the control program to be processed using the master and to transmita release to the slave via one of (i) a field bus and (ii) asynchronization connection in each case one of (i) after a predefinedinterval of time has expired and (ii) an event has occurred, the releaseindicating to the slave that it should process processing sections ofthe control program to be processed in a temporally asynchronous mannerusing the slave which correspond to those processing sections of thecontrol program to be processed by the master which have already beenprocessed up to the release.